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ABSTRACT 

An  Interactive  Project  Agreement  (IPA)  between  Electronic  Warfare  and  Radar 
Division  of  the  Defence  Science  Technology  Organisation  and  Thales  Australia 
(formerly  ADI  Limited)  has  an  initial  aim  to  evaluate  DSTO  innovations  flowing  from 
its  ELINT  Exploitation  for  Situational  Awareness  (ELEXSA)  task  within  Thales 
Australia's  operational  situational  awareness  tool  Llama-Cheetah.  This  document 
outlines  the  human  in  the  loop  experiment  undertaken  to  determine  whether  the  visual 
enhancements  provided  by  ELEXSA  computational  components  increased  operator 
situational  awareness,  thereby  improving  their  ability  to  achieve  tactical  goals. 
Analysis  of  three  hypotheses  showed  that  ELEXSA  Enhanced  Llama:  (1)  increases 
operator  survivability;  (2)  shortens  mission  duration;  and  (3)  reduces  the  time 
helicopter  spends  vulnerable  to  detection,  compared  to  Standard  Llama.  Analysis  of 
anecdotal  reports  showed  that  operators  using  ELEXSA  Enhanced  Llama  had  lower 
workload  and  stress  levels  and  more  accurate  perceptions  of  their  vulnerability  to 
radar  detection  than  operators  using  Standard  Llama. 
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An  Assessment  of  ELINT  Exploitation  for  Situational 
Awareness  Visualisations  on 
Operator  Situational  Awareness 


EXECUTIVE  SUMMARY 

An  Interactive  Project  Agreement  (IPA)  between  Electronic  Warfare  and  Radar 
Division  (EWRD)  of  the  Defence  Science  Technology  Organisation  (DSTO)  and  Thales 
Australia  (formerly  ADI  Limited)  was  agreed  in  September  2004.  An  initial  aim  of  the 
agreement  is  to  evaluate  DSTO  irmovations  flowing  from  its  ELINT  Exploitation  for 
Situational  Awareness  (ELEXSA)  task  within  Thales  Australia's  operational  situational 
awareness  tool  Llama-Cheetah. 

As  Llama-Cheetah  is  a  situational  awareness  tool  hosted  in  the  Joint  Command 
Support  Environment,  demonstrations  employing  Llama-Cheetah  have  a  technological 
fidelity  matching  existing  operational  capabilities.  Under  the  IPA,  Thales  Australia 
developed  two  versions  of  Llama-Cheetah:  Standard  Llama  and  ELEXSA  Enhanced 
Llama  which  incorporated  some  of  the  ELEXSA  computational  components. 

In  order  to  assess  the  effect  of  the  ELEXSA  visualizations  on  operator  situational 
awareness,  a  human  in  the  loop  experiment  was  undertaken  that  compared  the 
performance  of  one  group  of  operators  using  ELEXSA  Enhanced  Llama  against  the 
performance  of  another  group  of  operators  using  Standard  Llama.  Both  groups  of 
operators  were  tasked  with  achieving  the  same  tactical  goals  in  the  same  operational 
scenario. 

The  experiment  was  conducted  using  a  simulated  tactical  environment.  Each  operator 
was  required  to  navigate  a  helicopter  through  a  hostile  littoral  environment  containing 
enemy  radars  integrated  with  Man-Portable  Air  Defence  System  (MANPADS)  and 
then  to  land  the  helicopter  on  a  host  ship.  The  experiment  recorded  the  times 
associated  with  the  execution  of  the  helicopter's  mission,  including  times  when  the 
helicopter  was  within  the  detection  range  of  the  enemy  radars,  and  the  time  of  landing 
on  the  host  vessel.  The  experiment  also  recorded  any  failure  to  reach  the  host  ship  due 
to  either  being  shot  down  by  a  MANPADS  or  running  out  of  fuel.  Measurements  were 
analysed  to  determine  whether  the  ELEXSA  enhancements  improved,  degraded  or  had 
no  impact  on  the  operator's  awareness  of  threats  in  the  tactical  environment.  The 
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analysis  consisted  of  three  statistical  hypothesis  tests  and  interpretation  of  anecdotal 
reports  of  the  operator's  experience. 

Analysis  of  the  three  hypotheses,  at  95%  confidence  level,  showed  that  ELEXSA 
Enhanced  Llama:  (1)  increases  operator  survivability;  (2)  shortens  mission  duration; 
and  (3)  reduces  the  time  helicopter  spends  vulnerable  to  detection,  compared  to 
Standard  Llama. 

Analysis  of  interview  results  showed  that  operators  using  ELEXSA  Enhanced  Llama 
had  lower  workload  and  stress  levels  and  more  accurate  perceptions  of  their 
vulnerability  to  radar  detection  than  operators  using  Standard  Llama. 
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1.  Introduction 

An  Interactive  Project  Agreement  (IPA)  between  Electronic  Warfare  and  Radar 
Division  (EWRD)  of  the  Defence  Science  Technology  Organisation  (DSTO)  and  Thales 
Australia  (formerly  ADI  Limited)  was  agreed  in  September  2004.  An  initial  aim  of  the 
agreement  is  to  evaluate  DSTO  innovations  flowing  from  its  ELINT^  Exploitation  for 
Situational  Awareness  (ELEXSA)  task  within  Thales  Australia's  operational  situational 
awareness  tool  Llama-Cheetah. 

As  Llama-Cheetah  is  a  situational  awareness  tool  hosted  in  the  Joint  Command 
Support  Environment,  demonstrations  employing  Llama-Cheetah  have  a  technological 
fidelity  matching  existing  operational  capabilities.  Under  the  IPA,  Thales  Australia 
developed  two  versions  of  Llama-Cheetah:  Standard  Llama  and  ELEXSA  Enhanced 
Llama  which  incorporated  some  of  the  ELEXSA  computational  components. 

In  order  to  assess  the  effect  of  the  ELEXSA  visualizations  on  operator  situational 
awareness,  a  human  in  the  loop  experiment  was  undertaken  that  compared  the 
performance  of  one  group  of  operators  using  ELEXSA  Enhanced  Llama  against  the 
performance  of  another  group  of  operators  using  Standard  Llama.  Both  groups  of 
operators  were  tasked  with  achieving  the  same  tactical  goals  in  the  same  operational 
scenario. 

The  experiment  was  undertaken  between  November  2005  and  Eebruary  2006  and  the 
results  are  reported  in  this  document. 


2.  ELEXSA 

One  driver  for  DSTO's  ELEXSA  task  is  a  need  to  improve  the  survivability  of  ADE 
units  such  as  troop  transport  vessels  which  do  not  have  Electronic  Warfare  (EW)  threat 
warning  systems  or  EW  skilled  operators  but  still  need  timely  and  accurate 
information  about  potential  threats  in  their  tactical  environment  in  order  to  stay  in  safe 
locations. 

Under  the  ELEXSA  task,  EWRD  has  developed  knowledge  intensive  algorithms  to 
visualise  the  detection  capabilities  of  threat  radars  in  a  manner  which  can  be  easily 
understood  by  operators  without  specialist  EW  expertise.  These  algorithms  enrich 
ELINT  data  with  additional  information  such  as  technical  intelligence  in  order  to 
compute  detection  ranges  of  threat  systems  against  specific  vessels  or  aircraft. 


^  ELINT  Electronic  Intelligence 
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This  knowledge  intensive  process  comprises  two  stages  known  as  Object  Assessment 
(OA)  and  Situation  Assessment  (SA).  OA  is  a  sequential  information  enrichment 
process  whereby  a  signal  level  representation  of  a  sensed  entity  is  transformed  through 
emitter  and  platform  representations  into  a  capability  representation.  The  process  of 
information  enrichment  is  shown  in  Figure  1. 


Figure  1.  ELEXSA's  Process  of  Sequential  Enrichment. 


The  aim  of  SA  is  to  compute  relationships  of  interest  between  sensed  entities,  such  as 
the  detection  range  of  a  radar  against  a  specific  aircraft.  ELEXSA  SA  uses  attributes 
derived  in  ELEXSA  OA  (such  as  radar  cross  section  and  radiated  power)  and  domain 
knowledge  (such  as  propagation  physics)  to  compute  detection  ranges.  Eigure  2  shows 
a  visualisation  of  ELEXSA  SA. 
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Figure  2.  Sample  ELEXSA  Visualisation. 


3.  Llama 

The  Joint  Command  Support  System  (JCSS)  supports  the  staff  in  operational  or 
strategic  headquarters.  It  enables  commanders  to  manage  and  communicate  the 
complex  information  required  to  plan  and  execute  joint  force  operations  and  exercises. 
The  JCSS  communicates  with  other  Command  Support  Systems  as  well  as  civil  and 
other  Government  authorities  through  Defence  and  public  network  communications 
facilities. 

Llama  is  a  situation  monitoring  tool  within  the  JCSS  environment.  Llama  provides  the 
Joint  Forces  with  the  ability  to  manipulate  and  view  situational  data  on  a  geographic 
display,  as  well  as  to  manage  large  amounts  of  information  relating  to  the  positions 
and  movements  of  entities  in  any  selected  area  of  interest.  An  extendable  suite  of 
planning  and  command  decision  aids  is  available  within  Llama  to  support  analysis, 
monitoring  and  planning.  Llama  supports  a  networked  environment.  A  Llama  Internet 
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Server  manages  the  picture,  map,  scene,  overlay  and  template  libraries.  Connecting 
Llama  clients  are  able  to  use  a  set  of  common  libraries. 

The  track  data  displayed  in  Llama  is  managed  by  Cheetah.  Cheetah  is  a  server 
application  that  can  communicate  with  other  Cheetah  servers  and  other  command 
support  systems.  Cheetah  servers  can  be  connected  together  using  standard  PC 
networking  (TCP),  email  (SMTP)  or  serial  links  in  star,  web  or  serial  network 
topologies.  Cheetah  servers  perform  track  correlation,  fusion,  purging  and  persistent 
storage.  External  sources  can  pass  track  data  to  Cheetah  in  a  number  of  different 
formats;  standards  supported  include  OTH  Gold,  JUNIT,  ADFORM,  Link  11  and 
ADGESIT. 

Llama  Cheetah  can  be  setup  in  many  different  configurations;  Figure  3  shows  one 
example  of  a  Llama  Cheetah  configuration.  Three  Llama  clients  share  picture,  map, 
scene,  and  overlay  data  from  a  Llama  Internet  Server.  Four  Cheetah  servers  are 
connected  in  a  serial  topology. 


4.  Experimental  Infrastructure 

4.1  Infrastructure  Components 

The  experimental  environment  consisted  of  the  following  components: 

•  Situational  awareness  application  running  on  a  computer  with  screen, 
keyboard  and  mouse: 

o  ELEXSA  Enhanced  Llama  for  one  group;  and 
o  Standard  Llama  for  the  other  group; 

•  An  operational  scenario  that  exercises  the  capabilities  of  ELEXSA; 

•  Simulation  environment  that  enables  scenario  entities  to  be  modelled; 

•  Navigation  tool  for  steering  the  simulated  helicopter;  and 

•  Range  calculator  for  manually  calculating  radar  detection  ranges  based  on 
emitter  data  displayed  by  Standard  Llama. 
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Both  instantiations  of  Llama  were  instrumented  to  record  performance  data. 


Cheetah  Cheetah 


Che«lah  Tracks 


Cheetah 


Cheetah  Tracks 


Cheetah 


Figure  3.  Example  Llama/Cheetah  Network  Layout. 


4.2  FLEWSE 

DSTO  has  developed  a  synthetic  environment  known  as  the  Force  Level  Electronic 
Warfare  Synthetic  Environment  (ELEWSE)  to  help  in  the  development  of  concepts  of 
operation  for  EW  systems. 

A  schematic  of  the  ELEWSE  architecture  is  provided  in  Eigure  4  and  shows  the  main 
system  components.  Eor  the  purposes  of  the  experiment  described  in  this  document, 
these  are  the  Scenario  Toolkit  and  Generation  Environment  (STAGE)  and  Combined 
Sensor  Modelling  Infrastructure  (CSMl). 
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Components  are  connected  via  a  High  Level  Architecture  (HLA)  compliant 
framework.  A  benefit  of  HLA  is  that  it  provides  an  ability  to  connect  other  external 
HLA  compliant  models  into  FLEWSE  . 

As  well  as  modelling  the  motions  of  entities  such  as  aircraft,  ships  and  unmanned 
aerial  vehicles  (UAVs),  ELEWSE's  key  capability  is  modelling  the  electromagnetic 
interactions  between  these  entities. 


Figure  4.  Tire  FLEWSE  Architecture. 


4.3  Experimental  Developments 

To  connect  ELEWSE,  Cheetah  and  ELEXSA  a  software  module  was  developed,  the 
HLA  Bridge.  In  addition,  software  components  for  ELEXSA  OA,  ELEXSA  SA  and 
enabling  operator  control  of  the  helicopter  within  ELEWSE  were  also  developed. 

The  HLA  Bridge  subscribes  to  ELINT  data  provided  by  ELEWSE.  The  HLA  Bridge 
formats  the  ELINT  data  and  provides  it  to  ELEXSA  OA.  The  HLA  Bridge  also  obtains 
own-ship  and  friendly  force  tracks  from  ELEWSE,  translates  to  the  appropriate  format 
and  sends  to  a  Cheetah  server. 

The  ELEXSA  OA  software  receives  ELINT  data  from  the  HLA  Bridge  and  exploits 
ELEXSA  databases  to  yield  the  OA.  This  OA  is  sent  to  Cheetah  and  onward  to  the 
situation  monitoring  tool.  Llama.  A  schematic  of  this  set-up  is  shown  in  Eigure  5. 
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The  ELEXSA  Enhanced  Llama  software  was  modified  to  incorporate  the  computation 
and  display  of  enriched  information  provide  by  ELEXSA  SA.  The  changes  include: 


FLEWSE 


HLA  Bridge 


Decisions.' 

Asset  I 

/  Local 
Operator 


Simulated  Track 
kAesaages 


SHiHilaied 

Signal 

Representation 

MesMges 


Test  Environment 


ELEXSA 

Object 

Assessment 


Display 

Control 


Command  HQ 
Operator 


Command  HQ 
Operator 


System  Under  Test 


Figure  5.  Experimental  Set-up. 

•  Calculate  the  detection  range  of  each  radar  mode  against  own-ship  radar  cross 
section  using  propagation  physics  (it  is  assumed  that  the  altitude  of  the 
helicopter  does  not  change  and  terrain  shielding  is  not  taken  into  account). 

•  The  detection  range  of  hostile  emitters  is  automatically  displayed  to  the 
operator. 

•  Record  the  time  own-ship  is  within  detection  range  of  hostiles.  This  is  used  to 
measure  the  performance  of  the  operator. 

Eigure  6  shows  a  screenshot  of  the  ELEXSA  Enhanced  Llama  application  while  the 
human-in-the-loop  scenario  is  running. 

Eigure  7  shows  a  screenshot  of  standard  Llama  application  while  the  human-in-the- 
loop  scenario  is  running.  The  Standard  Llama  environment  does  not  automatically 
calculate  and  display  detection  range  information.  However,  several  graphical 
interfaces  were  developed  to  allow  the  operator  to  access  the  same  information  about 
detected  entities  as  was  embedded  within  ELEXSA  Enhanced  Llama.  Using  these 
interfaces  the  operator  must  first  enter  the  emitter  signature  into  the  detection  range 
calculator  (shown  in  Eigure  8)  to  obtain  a  value  for  the  maximum  detection  range  of 
the  emitter.  When  the  operator  has  a  value  for  the  detection  range  of  the  emitter,  they 
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may  manually  draw  circles  in  the  situation  display.  The  detection  range  calculator  is 
available  only  in  the  Standard  Llama  environment  (as  detection  ranges  are 
automatically  calculated  and  displayed  in  the  Enhanced  Llama  environment). 
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Figure  7.  The  Standard  Llama  Visualisation. 
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Figure  8.  The  Detection  Range  Calculator. 


Software  tools  to  allow  control  of  the  helicopter  and  awareness  of  helicopter  fuel 
reserves  were  also  developed: 

•  The  Compass  Controller  to  allow  the  Operator  to  control  the  heading  of  the 
helicopter.  The  operator  cannot  change  the  speed  or  altitude  of  the  helicopter. 
A  screenshot  of  the  compass  controller  is  shown  in  Figure  9. 

•  The  fuel  remaining  indicator  displays  how  minutes  of  fuel  remain.  It  counts 
down  from  15  minutes.  A  screenshot  of  the  fuel  remaining  indicator  is  shown 
in  Figure  10. 

Both  of  these  tools  were  available  in  both  Llama  environments. 
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Figure  9.  The  Compass  Tool. 


Figure  10.  The  Fuel  Remaining  Indicator. 


5.  Methodology 


5.1  Overview 

The  experiment  measured  the  performance  of  the  two  groups  of  operators  in  achieving 
the  same  tactical  goals  within  the  same  operational  scenario,  using  the  two  different, 
but  functionally  equivalent  situation  assessment  tools:  Standard  and  Enhanced  Llama- 
Cheetah. 

Two  groups  of  operators  conducted  the  experiment;  enhanced  Llama  operators  and 
standard  Llama  operators. 
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1.  Enhanced  Llama  operators  were  provided  with  an  automatic  visualisation  for 
the  detection  range  of  hostile  radars.  When  the  EW  signature  was  available,  the 
enhanced  Llama  calculated  the  detection  range  using  ELEXSA  and  displayed 
the  range  as  a  blue  semi-transparent  ring  as  shown  above  in  Eigure  6. 

2.  Standard  Llama  operators  did  not  have  the  automatic  visualisation  of  the 
detection  range  of  hostile  radars.  Instead,  they  were  required  to  enter  the  EW 
signature  into  the  detection  range  calculator  (Eigure  8)  and  then  use  the 
drawing  tools  available  within  Llama  to  draw  the  detection  ranges,  as  shown 
above  in  Eigure  7. 

5.2  Operational  Scenario 

The  experiment  was  conducted  using  a  simulated  tactical  environment.  Each  operator 
was  required  to  navigate  a  Black  Hawk  helicopter  around  an  archipelago  nation  that 
had  recently  been  over-run  by  a  hostile  force  and  successfully  land  the  helicopter  on  its 
host  ship. 

The  hostile  force  had  installed  several  Big  Bird  radar  surveillance  systems  integrated 
with  several  Snowdrift  missile  systems  and  many  MANPADS  capable  combat  teams  to 
protect  itself  from  attack.  The  hostile  force  also  has  two  naval  vessels:  a  patrol  boat 
with  Don-2  radar  and  one  Sovremenny  class  vessel.  Both  vessels  patrol  near  the  coast. 

Eorward  of  the  helicopter  was  a  number  of  Unmanned  Air  Vehicles  (UAVs)  equipped 
with  ELINT  sensors.  Detections  from  these  sensors  were  transmitted  to  the  helicopter 
for  interpretation  and  visualisation  on  its  situational  awareness  display. 

At  the  beginning  of  the  scenario,  the  helicopter  had  almost  completed  a  circuit  of  the 
archipelago;  there  was  one  more  island  to  cross  to  reach  its  host  ship.  There  was 
approximately  15  minutes  of  fuel  remaining  onboard  the  helicopter,  so  to  successfully 
complete  the  mission  with  the  allotted  amount  of  fuel;  the  operator  was  required  to 
traverse  the  last  island  rather  than  circumnavigating  it.  The  operator  was  aware  of  the 
hostile  force  and  its  capabilities. 

The  goal  of  the  operator  was  to  avoid  detection  by  the  radar  systems,  possible 
destruction  by  the  MANPADS  and  land  on  the  ship  before  rurming  out  of  fuel.  If  the 
helicopter  loitered  in  a  region  where  it  was  vulnerable  to  detection  by  a  radar  for  more 
than  30  seconds,  for  the  purposes  of  this  experiment,  it  was  assumed  that  the  radar 
system  detected  the  helicopter  and  reported  its  location  to  a  MANPADS  in  the  vicinity 
with  lethal  consequences  (i.e.  Puii  =  100%  if  the  helicopter  was  within  the  radar's 
detection  range  for  more  than  30  seconds).  The  simulation  was  played  at  faster  than 
real  time  to  enable  a  tractable  experiment  to  be  conducted  within  a  reasonable  amount 
of  the  participant's  time. 
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5.3  Data  Acquisition 

During  the  experiment,  the  following  information  was  recorded  for  each  operator: 

•  The  time  taken  for  the  helicopter  to  reach  its  host  ship  (if  the  mission  was 
successful); 

•  The  times  when  the  helicopter  was  within  the  detection  range  of  any  of  the 
hostile  radars;  and 

•  Any  failure  to  reach  the  host  ship  due  to  either  being  shot  down  by  a  MANPAD 
or  rurming  out  of  fuel. 

After  each  operator  had  completed  their  mission,  they  were  asked  a  small  number  of 
questions  to  solicit  their  perceptions  of: 

•  Their  workload  and  stress  level; 

•  The  amount  of  time  they  felt  vulnerable  to  detection; 

•  The  utility  of  the  tools 

5.4  Analysis 

Statistical  hypothesis  testing  was  used  to  determine  whether  the  ELEXSA 
enhancements  improved,  degraded  or  had  no  impact  on  the  operator's  awareness  of 
threats  in  their  tactical  environment.  In  statistical  hypothesis  testing,  the  question  of 
interest  is  simplified  into  two  competing  hypotheses  between  which  we  have  a  choice; 
the  null  hypothesis,  denoted  Ho,  and  the  alternative  hypothesis,  denoted  Hi.  The 
experiments  are  then  carried  out  in  an  attempt  to  disprove  the  null  hypothesis.  The 
null  hypothesis  cannot  be  rejected  unless  the  evidence  against  it  is  sufficiently  strong. 
This  is  due  to  the  fact  that  the  null  hypothesis  relates  to  the  statement  being  tested, 
whereas  the  alternative  hypothesis  relates  to  the  statement  to  be  accepted  if  the  null  is 
rejected.  The  final  conclusion  once  the  test  has  been  carried  out  is  always  given  in 
terms  of  the  null  hypothesis.  We  either  'reject  Ho  in  favour  of  Hi'  or  'do  not  reject  Ho'; 
we  never  conclude  'reject  Hi',  or  even  'accept  Hi'.  If  we  conclude  'do  not  reject  Ho',  this 
does  not  necessarily  mean  that  the  null  hypothesis  is  true,  it  only  suggests  that  there  is 
not  sufficient  evidence  against  Ho  in  favour  of  Hi;  rejecting  the  null  hypothesis  then, 
suggests  that  the  alternative  hypothesis  may  be  true. 

The  significance  level  of  a  statistical  hypothesis  test  is  a  fixed  probability  of  wrongly 
rejecting  the  null  hypothesis  Ho,  if  it  is  in  fact  true.  That  is,  the  significance  level  is 
made  as  small  as  possible  in  order  to  protect  the  null  hypothesis.  We  have  chosen  a 
significance  level  of  0.05  =  5%. 

The  critical  value(s)  for  a  hypothesis  test  is  a  threshold  to  which  the  value  of  the  test 
statistic  in  a  sample  is  compared  to  determine  whether  or  not  the  null  hypothesis  is 
rejected. 
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The  critical  value  for  any  hypothesis  test  depends  on  the  significance  level  at  which  the 
test  is  carried  out,  and  whether  the  test  is  one-sided  or  two-sided. 

The  three  hypotheses  developed  to  determine  the  effect  of  the  ELEXSA  enhancements 
on  operator's  threat  awareness  were: 

•  Hypothesis  1: 

Hi  =  Use  of  Enhanced  Llama-Cheetah  results  in  better  operator 
survivability  than  Standard  Llama-Cheetah 

Ho  =  Use  of  Enhanced  Llama-Cheetah  does  not  result  in  better  survivability 
than  standard  Llama 

In  order  to  test  this  hypothesis,  the  average  number  of  operators  safely 
landing  the  helicopter  on  the  ship  is  tallied  for  both  Standard  Llama- 
Cheetah  (B_finished)  and  Enhanced  Llama-Cheetah  (E_finished).  If  there  is 
not  sufficient  evidence  against  Ho  in  favour  of  Hi,  the  difference  between 
the  numbers  of  operators  successfully  completing  the  mission  will  not  be 
statistically  significant. 

•  Hypothesis  2: 

Hi  =  Use  of  Enhanced  Llama-Cheetah  results  in  shorter  mission  durations 
than  Standard  Llama-Cheetah 

Ho  =  Use  of  enhanced  Llama-Cheetah  does  not  result  in  shorter  mission 
durations  than  Standard  Llama-Cheetah 

In  order  to  test  this  hypothesis,  the  average  time  taken  by  both  groups  of 
operators  to  safely  land  the  helicopter  on  the  ship  is  calculated  for  both 
Standard  Llama-Cheetah  (B_time_taken)  and  Enhanced  Llama-Cheetah 
(E_time_taken).  If  there  is  not  sufficient  evidence  against  Ho  in  favour  of 
Hi,  the  difference  between  the  average  times  taken  to  successfully  complete 
the  mission  will  not  be  statistically  significant. 

•  Hypothesis  3: 

Hi  =  Use  of  Enhanced  Llama-Cheetah  results  in  a  reduction  in  the  time  the 
Helicopter  is  venerable  to  detection  than  Standard  Llama-Cheetah 
Ho  =  Use  of  Enhanced  Llama-Cheetah  does  not  results  in  a  reduction  in  the 
time  the  Helicopter  is  venerable  to  detection  than  Standard  Llama-Cheetah 
In  order  to  test  this  hypothesis,  the  average  time  spent  by  both  groups  of 
operators  helicopters  within  the  detection  ranges  of  hostile  radars  is 
calculated  for  both  Standard  Llama-Cheetah  (B_seen)  and  Enhanced 
Llama-Cheetah  (E_seen).  If  there  is  not  sufficient  evidence  against  Ho  in 
favour  of  Hi,  the  difference  between  the  average  times  spent  by  operators 
within  the  detection  ranges  of  hostile  radars  will  not  be  statistically 
significant. 

In  addition  to  the  statistical  hypothesis  testing,  the  operator's  anecdotal  perception  of 
their  own  vulnerability  was  compared  against  the  ground  truth.  The  ground  truth 
measurements  were  obtained  from  log  files  of  the  relative  position  of  the  helicopter 
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and  threat  radars.  The  operator  reports  on  their  perception  of  their  workload  were  also 
compared  for  the  two  Llama-Cheetah  environments.  Interview  anecdotes  were 
mapped  into  high,  medium  or  low  measures. 


6.  Conduct  of  the  Experiment 


6.1  Personnel 

The  following  personnel  assisted  in  the  conduct  of  the  experiment: 

•  Coordinators.  Responsible  for  briefing  operators  and  co-ordinating  each 
scenario  execution;  and 

•  Operators.  Responsible  for  executing  the  scenarios.  Operators  were  randomly 
assigned  from  a  pool  of  16  into  two  experimental  groups  of  8  members  each. 
Eight  operators  were  chosen  from  both  Thales  Australia  and  DSTO.  The  chosen 
operators  did  not  have  detailed  knowledge  of  Llama  or  ELEXSA  and  as  such, 
they  required  some  training  in  the  use  of  Llama  (outlined  in  Section  6.2).  Most 
operators  had  some  expertise  with  software  and  modelling  and  some  had 
military  expertise  from  previous  work  roles. 

6.2  Training 

Before  their  mission  began,  each  operator  was  provided  with  training  by  a  coordinator. 
During  their  training,  each  operator  was  given  an  induction  which  explained  the 
experimental  aims  and  the  operational  scenario.  Operators  were  then  provided  with 
some  training  in  the  use  of  Llama-Cheetah.  This  part  of  the  training  consisted  of  two 
phases:  a  PowerPoint  overview  of  the  capabilities  of  Llama,  and  a  hands-on  period  to 
gain  practice  in  the  tasks  required. 

All  the  operators  were  trained  in: 

•  How  to  interpret  icons  on  the  screen; 

•  How  to  interrogate  icons  on  the  screen; 

•  How  to  measure  fuel  remaining; 

•  How  to  steer  the  helicopter; 

•  How  to  land  the  helicopter;  and 

•  How  to  use  zoom  and  pan  buttons. 

Operators  using  Standard  Llama  were  also  trained  in: 

•  How  to  use  the  detection  range  calculator; 

•  How  to  find  the  distance  between  two  clicked  points;  and 

•  How  to  overlay  a  circle  with  a  desired  radius  on  the  screen. 
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After  the  training  was  completed,  operators  were  given  a  representative  Llama  task  to 
complete.  Had  an  operator  been  unable  to  perform  the  required  task,  the  training 
would  have  been  repeated,  however  this  did  not  occur. 

6.3  Conduct  of  Serials 

As  the  operator  came  "on  duty",  the  experiment  coordinators  checked  that  they  were 
ready  and  started  the  simulation.  The  operator  clicked  the  time  clock  to  start  the 
session.  The  operator  then  responded  to  information  presented  by  Llama  until  the 
mission  was  successfully  completed  or  the  15  minute  fuel  limit  was  reached.  At  the 
end  of  the  run,  the  coordinators  interviewed  the  operator  using  a  standard  set  of 
questions  in  order  to  obtain  additional  information  regarding  their  experience.  After 
each  operator  had  completed  their  session,  the  experiment  coordinator  verified  that  the 
experimental  infrastructure  performed  as  expected,  that  the  flight  log  file  had  been 
correctly  written  and  archived,  and  that  other  records  identifying  the  operator's  start 
and  end  times  had  also  been  written.  In  all  cases  this  process  was  successfully 
completed. 

6.4  Experimental  Serials 

In  all,  16  serials  were  completed,  8  for  Standard  Llama  and  8  for  ELEXSA  Enhanced 
Llama. 


7.  Observations 

7.1  Mission  Completion  and  Vulnerability  Data 

The  mission  time  and  time  vulnerable  to  detection  for  each  operator  is  shown  below. 
Table  1  shows  the  results  for  operators  using  Standard  Llama.  Table  2  shows  the 
results  for  operators  using  ELEXSA  Enhanced  Llama.  Mission  durations  of  900  s 
indicate  that  the  operator  ran  out  of  fuel  and  crashed. 
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Table  1.  Results  for  operators  using  Standard  Llama.  Snow  drift,  Don-2  and  Big  Bird  were  the 
threat  radar  systems  used  during  the  experiment. 


Operator 

Number 

Mission 

Duration 

(s) 

Time  Vulnerable  to  Radar  Detection  (s) 

SNOW 

DRIET 

DON-2 

BIG  BIRD 

BIG  BIRD 

Total 

1 

900.000 

0 

0 

0 

0 

0 

2 

702.994 

0 

232.721 

114.356 

0 

347.077 

3 

900.000 

11.235 

51.703 

91.546 

0 

154.484 

4 

900.000 

0 

0 

182.848 

122.646 

305.494 

5 

608.825 

0 

0 

0 

0 

0 

6 

762.151 

0 

0 

14.540 

0 

14.540 

7 

704.833 

0 

48.204 

46.891 

0 

95.095 

8 

900.000 

0 

0 

0 

0 

0 

For  operators  using  Standard  Llama,  the  average  time  taken  to  reach  the  destination 
was  797.350  seconds  and  the  standard  deviation  was  117.346  seconds. 


Table  2.  Results  for  operators  using  ELEXSA  Enhanced  Llama.  Snow  drift,  Don-2  and  Big 
Bird  were  the  threat  radar  systems  used  during  the  experiment. 


Operator 

Number 

Mission 

Duration 

(s) 

Time  Vulnerable  to  Radar  Detection  (s) 

SNOW 

DRIET 

DON-2 

BIG  BIRD 

BIG  BIRD 

Total 

1 

572.704 

0 

0 

0 

0 

0 

2 

546.179 

0 

0 

0 

0 

0 

3 

543.275 

0 

0 

0 

0 

0 

4 

557.047 

0 

0 

0 

0 

0 

5 

535.083 

0 

0 

0 

0 

0 

6 

564.426 

0 

0 

0 

0 

0 

7 

557.107 

0 

0 

0 

0 

0 

8 

572.751 

0 

0 

0 

0 

0 

For  operators  using  ELEXSA  Enhanced  Llama,  the  average  time  taken  to  reach  the 
destination  was  556.072  seconds  and  the  standard  deviation  was  13.778  seconds. 


7.2  Interview  Results 

The  interview  records  are  included  as  Appendix  1.  These  data  have  been  collated  into 
Tables  3  and  4  below.  Table  3  shows  data  for  operators  using  standard  Llama.  Table  4 
shows  data  for  operators  using  ELEXSA  Enhanced  Llama. 

In  both  tables  the  operators  perception  of  their  vulnerability  to  any  radars  (V  = 
perceived  to  be  vulnerable,  U  =  undecided,  N  =  perceived  not  to  be  vulnerable)  is 
shown  together  with  the  experimental  measured  vulnerability  (V  =  Vulnerable,  N  = 
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Not  Vulnerable).  In  addition  an  assessment  of  the  operator's  perceived  workload  and 
stress  level  as  high  medium  or  low  is  shown. 


Table  3.  Interview  results  for  operators  using  Standard  Llama. 


Opl 

Op  2 

Op  3 

Op  4 

Op  5 

Op  6 

Op  7 

Op  8 

Perceived 
to  be 

Vulnerable 

N 

V 

U 

N 

V 

N 

V 

V 

Measured 
to  be 

Vulnerable 

N 

V 

V 

V 

N 

V 

V 

N 

Workload 
and  Stress 
Level 

Low 

High 

Med 

High 

Med 

Low 

High 

High 

Table  4.  Interview  results  for  operators  using  ELEXSA  Enhanced  Elama. 


Opl 

Op  2 

Op  3 

Op  4 

Op  5 

Op  6 

Op  7 

Op  8 

Perceived 
to  be 

Vulnerable 

N 

N 

N 

N 

N 

N 

N 

N 

Measured 
to  be 

Vulnerable 

N 

N 

N 

N 

N 

N 

N 

N 

Workload 
and  Stress 
Level 

Low 

Low 

Low 

Low 

Med 

Low 

Low 

Low 

8.  Analysis 

8.1  Analysis  Techniques  and  Definitions 

To  test  Hypothesis  1,  the  Fisher  Exact  Probability  Test  was  used.  This  test  is 
overviewed  in  Appendix  2  and  is  described  in  [1] . 

To  test  Hypotheses  2  and  3,  the  two  sided  t-test  was  used.  This  test  is  overviewed  in 
Appendix  3  and  is  described  in  [1] . 

The  definition  of  survivability  used  in  this  experiment  is  of  completing  the  mission  by 
landing  on  the  host  vessel.  There  are  two  sources  of  mission  failure: 
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•  Being  shot  down ;  and 

•  Running  out  of  fuel. 

In  the  experimental  scenario,  being  in  the  detection  zone  of  any  radar  for  more  than  30 
seconds  was  postulated  as  enough  time  for  the  helicopters  position  to  be  determined 
and  disseminated  to  a  MANPADS  in  the  vicinity  with  a  lethal  outcome.  Given  that 
ELEXSA  is  about  survivability  against  radar  threats  in  a  tactical  mission,  survivability 
in  this  experiment  is  measured  as: 

•  Successfully  landing  the  helicopter  on  the  parent  ship;  and 

•  Not  being  within  the  detection  range  of  a  radar  for  more  than  30  seconds. 

8.2  Hypothesis  1 

In  the  first  hypothesis,  the  aim  is  to  test  whether  any  difference  in  the  proportion  of 
Standard  Llama  operators  completing  the  assigned  mission  and  the  proportion  of 
Enhanced  Llama  operators  completing  the  assigned  mission  is  statistically  significant. 
The  data  presented  in  section  7  was  used  in  a  Eisher  Exact  Probability  Test. 

A  2x2  contingency  table  (as  described  in  Appendix  2)  used  to  express  the  relationship 
between  the  variables  for  the  ELEXSA  HIE  Experiment's  first  hypothesis  is  shown  in 
Table  5. 


Table  5.  The  2x2  contingency  table  representing  the  proportion  of  operators  successfully 
completing  the  mission  for  both  Standard  and  Enhanced  Llama-Cheetah  environments. 


Standard 

Llama 

Enhanced 

Llama 

Total 

Completed 

2 

8 

10 

Not 

completed 

4 

0 

4 

Total 

6 

8 

14 

Pcutoff  has  been  computed  for  this  experiment  to  be  0.003. 

By  summing  the  probabilities  of  all  other  matrices  with  the  same  marginal  totals  and 
sample  size  which  are  less  than  or  equal  to  the  Pcuttoff,  a  two  sided  probability  of  0.015  is 
obtained.  Since  this  value  is  less  than  the  significance  level  of  0.05,  the  proportion  of 
operators  successfully  completing  the  assigned  mission  is  statistically  significant  i.e. 
there  is  a  statistically  significant  association  between  the  type  of  Llama  used  by  the 
operator  and  the  operator  successfully  completing  the  mission.  Thus  the  null 
hypothesis  can  be  rejected  and  Hypothesis  1  (at  95%  level)  can  be  accepted. 

Note  that  there  were  two  operators  who  ran  out  of  fuel  but  stayed  outside  detection 
ranges.  These  operators  were  very  cautious.  These  2  operators  were  excluded  from  the 
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analysis  based  on  the  adopted  definition  of  survivability  and  so  are  not  included  in 
Table  5.  If  these  two  operators  are  included  in  the  analysis,  since  they  failed  to 
complete  the  mission,  a  two  sided  probability  of  0.007  is  obtained,  which  is  less  than 
the  significance  level  of  0.05.  Thus  the  null  hypothesis  can  be  rejected  and  Hypothesis 
1  (at  95%  level)  can  be  accepted  whether  or  not  these  operators  are  included  in  the 
analysis. 

8.3  Hypothesis  2 

For  the  second  hypothesis,  a  f-test  [2]  was  used.  For  both  instantiations  of  Llama,  the 
average  time  taken  to  return  the  helicopter  to  the  ship  was  measured.  All  16  operators 
were  used  in  this  analysis  with  those  that  ran  out  of  fuel  taking  15  minutes.  The  critical 
t  value  for  14  degrees  of  freedom  and  p  =  0.05  (ie  95%  confidence  level)  =  2.145.  The 
calculated  value  of  t  was  5.78  and  since  it  is  larger  than  the  critical  value,  the  null 
hypothesis  can  be  rejected  and  Hypothesis  2  (at  95%  level)  can  be  accepted. 

8.4  Hypothesis  3 

For  the  third  hypothesis,  a  f-test  was  used.  For  both  instantiations  of  Llama,  the 
average  time  spent  in  "may  be  seen"  regions  of  the  threats  was  computed.  All  16 
operators  were  used  in  this  analysis.  The  critical  t  value  for  14  degrees  of  freedom  and 
p  =  0.05  (ie  95%  confidence  level)  =  2.145.  The  calculated  value  of  t  was  2.28  and  since 
it  is  larger  than  the  critical  value,  the  null  hypothesis  can  be  rejected  and  Hypothesis  3 
(at  95%  level)  can  be  accepted. 

8.5  Analysis  of  Interviews 

Consideration  of  the  results  presented  in  Tables  3  and  4  allows  the  following 
assessments. 

(1)  The  perceptions  of  operators  using  ELEXSA  Enhanced  Llama  about  their 
vulnerability  to  radar  detection  were  accurate. 

(2)  The  perceptions  of  operators  using  Standard  Llama  about  the  vulnerability  to 
radar  detection  was  not  reliable  with  only  3  operators  perceptions  matching 
measured  vulnerability.  3  operators  thought  they  were  not  detected  when  they 
were  and  two  operators  thought  they  were  detected  when  they  were  not. 

(3)  The  median  reported  work  load  and  stress  level  of  operators  using  ELEXSA 
Enhanced  Llama  was  "low"  whereas  for  those  using  Standard  Llama  was 
"high". 

8.6  Sample  Size 

As  mentioned  in  Section  6.1,  two  groups  of  operators  were  used  in  the  conduct  of  the 
experiment.  Each  group  of  operators  comprised  8  members. 

The  data  presented  in  Section  7  included  computations  of  mean  and  standard 
deviation.  Eor  operators  using  Standard  Llama,  the  average  time  taken  to  reach  the 
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destination  was  797.350  seconds  and  the  standard  deviation  was  117.346  seconds.  For 
operators  using  ELEXSA  Enhanced  Llama,  the  average  time  taken  to  reach  the 
destination  was  556.072  seconds  and  the  standard  deviation  was  13.778  seconds.  There 
is  no  overlap  between  the  ranges  of  the  standard  deviations  of  these  two  sets  of 
observations  and  so  this  suggests  that  the  sample  is  large  enough  to  reduce  the  effects 
of  biases  caused  by  operator  competence  levels  for  this  experiment.  Analysis 
immediately  above  shows  that  the  results  were  significant  at  the  95%  confidence  level, 
indicating  that  the  sample  size  is  adequate. 

Had  additional  experimental  runs  been  undertaken  it  seems  reasonable  to  expect  that 
the  result  would  be  to  reduce  the  standard  deviation  of  the  observations,  particularly 
of  the  operators  using  Standard  Llama.  As  additional  experimental  runs  would  incur 
additional  costs  and  the  sample  size  of  two  sets  of  8  operators  is  adequate,  it  was 
decided  not  to  undertake  additional  experimental  runs. 


9.  Conclusions 

This  document  has  described  the  results  of  an  experiment  undertaken  under  an 
Interactive  Project  Agreement  between  DSTO  and  Thales  Australia. 

The  experiment  measured  the  performance  of  two  groups  of  operators  in  achieving  the 
same  tactical  goals  in  the  same  scenario  using  two  different  but  functionally  equivalent 
tool  sets  based  on  the  Situational  Awareness  application  Llama.  In  ELEXSA  Enhanced 
Llama,  information  about  detection  ranges  is  always  computed  and  displayed.  In 
Standard  Llama,  operators  may  choose  to  compute  and  display  this  information. 

The  experiment  was  conducted  using  a  simulated  tactical  environment.  Each  operator 
was  required  to  navigate  a  helicopter  through  a  hostile  environment  containing  enemy 
radars  integrated  with  MANPADS  to  land  the  helicopter  on  a  host  ship.  The 
experiment  recorded  the  times  associated  with  the  execution  of  the  helicopter's 
mission,  including  times  when  the  helicopter  was  within  the  detection  range  of  the 
enemy  radars,  and  the  time  of  landing  on  the  host  vessel.  The  experiment  also 
recorded  any  failure  to  reach  the  host  ship  due  to  either  being  shot  down  by  a 
MANPADS  or  rurming  out  of  fuel.  Measurements  were  analysed  to  determine  whether 
the  ELEXSA  enhancements  improved,  degraded  or  made  no  impact  on  operator 
situational  awareness  of  threats  in  the  tactical  environment.  The  analysis  consisted  of 
three  statistical  hypothesis  tests  and  assessment  of  anecdotal  reports  of  the  operator's 
experience. 

Hypothesis  test  analyses,  at  95%  confidence  level,  showed  that  ELEXSA  enhanced 
Llama:  (1)  increases  operator  survivability  (2)  shortens  mission  duration,  and  (3) 
reduces  the  time  helicopter  vulnerable  to  detection,  compared  to  Standard  Llama. 
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Analysis  of  interview  results  showed  that  operators  using  ELEXSA  Enhanced  Llama 
had  lower  workload  and  stress  levels  and  more  accurate  perceptions  of  their 
vulnerability  to  radar  detection  than  operators  using  Standard  Llama. 
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Appendix  A:  Operator  Questionnaire 


Answers  to  Questions  for  Standard  Lama  Operator  1 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No.  With  the  software,  I  can  avoid  radars.  I  could  have  taken  a  risk  and  attempted  to  navigate 
through  the  small  gap.  I  decided  it  was  safer  to  fly  around,  but  ran  out  of  fuel. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

The  detection  range  circle  moves  when  the  ships  move. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

The  detection  range  circle  moves  when  the  ships  move. 

Summary:  Operator  1  didn't  get  detected.  Operator  1  didn't  reach  destination. 

Answers  to  Questions  for  Standard  Lama  Operator  2 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

Yes.  I  wondered  into  the  detection  range  at  the  start.  Helicopter  was  moving  too  fast. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Using  manual  checkup.  Yes. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

No.  It  is  very  difficult  at  the  start.  The  scenario  is  not  real  time. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

An  audio  alert  when  a  new  detection  is  discovered. 

Do  you  feel  the  tools  were  useful  and  useable? 

Useful,  yes.  But  complicated  to  use.  There  a  multiple  tasks  to  do  at  once.  Two  operators  (a 
navigator)  would  help. 

What  could  be  changed  to  improved  useability? 

Could  be  a  automated;  circles  drawn  automatically.  A  database  lookup  to  calculate  detection 
ranges. 

Not  to  scale.  Difficult  to  determine  distance 

A  lookup  table  on  a  piece  of  paper  would  help.  Lists  likely  threads  and  detection  ranges. 
Radius  of  circles  displayed  during  circle  created,  (displayed  when  dragging). 
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Summary:  Operator  2  was  detected  three  times.  Once  by  Big  Bird,  twice  by  DON-2.  Reached 
destination  in  about  12:00  mins. 

Answers  to  Questions  for  Standard  Lama  Operator  3 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

I  was  on  the  peripheral  of  some  radars,  on  the  borderline. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes.  Some  people  can  react  faster  than  others.  With  more  practice,  my  reaction  would  be 
faster. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

I  needed  more  practice. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

A  touch  screen  with  big  square  buttons. 

Summary:  Operator  3  was  detected  three  times  by  Big  Bird,  DON-2  and  snowdrift.  Operator  3 
didn't  reach  the  destination. 

Answers  to  Questions  for  Standard  Lama  Operator  4 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No,  of  course. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

I  was  overloaded  at  the  beginning.  The  task  was  cumbersome. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

Away  to  identify  hostiles  that  are  the  same.  Hostiles  that  have  the  same  emitter  use  the  same 
colour,  etc. 

Do  you  feel  the  tools  were  useful  and  useable? 

Useful  yes.  Maybe  not  useable. 

What  could  be  changed  to  improved  useability? 

Once  hostiles  detected  the  circles  automatically  appear.  If  the  hostile  moves,  the  circle  moves 
with  it. 

Summary:  Operator  4  was  detected  by  Big  Bird  1  and  by  Big  Bird  2.  Operator  4  didn't  reach 
destination.  Splash  down  130km  WSW  of  ship. 
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Answers  to  Questions  for  Standard  Lama  Operator  5 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

Yes,  by  one  of  the  moving  ships.  It  looked  like  they  may  cross. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Probably. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

No. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

I  would  rather  hit  'enter'  than  'apply'  when  adjusting  circles  radius. 

Circles  should  follow  platform  movements.  When  you  have  lots  to  do,  moving  circles  is 
cumbersome. 

Calculated  and  Llama  should  be  integrated. 

Summary:  Operator  5  didn't  get  detected  and  arrived  at  the  destination  in  10:15  minutes. 

Answers  to  Questions  for  Standard  Lama  Operator  6 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No.  I  had  a  clear  picture.  Only  limitation  was  my  own  mistake  at  the  beginning. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes.  The  challenge  is  just  using  it  correctly. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes.  It  is  important  to  prioritise  your  time. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

Circles  created  around  hostiles  automatically.  Provide  a  warning  rather  than  letting  you  wonder 
into  detection. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

The  interface  is  cluttered.  Buttons  and  button  position  should  be  simplified. 

Summary:  Operator  6  was  detected  by  Big  Bird  2  and  reached  destination  in  approx.  13:15 
mins. 

Answers  to  Questions  for  Standard  Lama  Operator  7 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 
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Yes,  based  on  the  provided  range  calculation. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

How  much  do  you  need  to  know?  How  accurate  is  the  intel? 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

No,  not  at  that  speed.  If  the  scenario  was  real  time  it  would  have  been  better. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

As  the  system  can  ID  the  radar  types,  the  circles  should  be  created  automatically.  The  circle 
should  auto  follow  moving  vessels. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes.  Symbology  needs  improving. 

What  could  be  changed  to  improved  useability? 

Someway  of  alerting  the  user  of  mobile  hostiles. 

Summary:  Operator  7  was  detected  by  Big  Bird  2  and  DON-2  twice  and  reached  destination  in 
approx.  11:50  mins. 

Answers  to  Questions  for  Standard  Lama  Operator  8 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

Yes,  one  of  the  frigates.  I  was  very  close. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

No.  If  I  had  more  time  I  would  have  taken  a  different  route. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

A  scale  on  the  map.  Some  indication  of  the  detection  range  of  UAVs. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

Double  click  to  zoom  out.  Drag  box  should  zoom  in  (currently  it  selects  targets  within  box). 

Summary:  Operator  8  was  not  detected  by  any  radars.  Operator  8  failed  to  reach  the 
destination. 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  1 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No.  If  the  circles  are  correct. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Funny  question.  I  think  the  information  is  correct.  If  I  get  'pinged'  by  a  hostile,  I  want  to  know 
what  to  do. 


26 


DSTO-TR-1924 


Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

No. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

Ability  to  return  to  the  previous  zoom  level. 

Two  views  of  the  world;  A  zoomed  in  view  and  a  'big  picture'  view. 

Summary:  Operator  1  didn't  get  detected.  Reached  destination  in  about  8:30  mins. 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  2 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No.  Not  according  to  the  information  provided. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes.  I  thought  'how  reliable  is  the  information?' 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

No.  I  was  worried  something  (a  radar  detection)  would  pop-up. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

The  circles  are  blue.  Blue  indicates  friendly.  Perhaps  should  be  red,  but  that  might  obscure  the 
icon. 

Summary:  Operator  2  didn't  get  detected  and  reached  destination  in  approx  9:00  mins. 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  3 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No.  Based  on  the  information  provided. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes.  There  is  never  enough  information.  Enough  information  to  determine  a  safe  route. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

If  the  detection  occurred  earlier.  I  can  foresee  a  situation  where  you  could  be  trapped  in  a 
location  by  hostile  radars. 
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Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  Improved  useablllty? 

Mouse  control  of  helicopter  instead  of  compass. 

Infinitely  scroll  with  scroll  bars  rather  than  using  the  pan  button. 

Summary:  Operator  3  wasn't  detected  and  reached  destination  in  about  9:00  mins. 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  4 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No. 

Did  you  feel  that  you  had  enough  Information  about  the  detected  radars? 

Yes. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes.  The  detections  were  early  enough. 

Is  there  additional  Information  that  could  have  made  your  job  easier? 

No.  I  wanted  to  stay  away  from  SAM  sites.  If  I  was  going  to  fall  into  a  detection  range,  I'd  prefer 
it  to  be  a  RADAR  site. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  Improved  useablllty? 

Not  really.  It  is  a  nice  setup. 

Summary:  Operator  4  didn't  get  detected  and  reached  destination  in  approx  9:30  mins. 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  5 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

Might  have  come  close.  No.  I  am  relying  on  the  information  being  accurate. 

Did  you  feel  that  you  had  enough  Information  about  the  detected  radars? 

Yes. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes.  It  was  tight  with  the  moving  ships. 

Is  there  additional  Information  that  could  have  made  your  job  easier? 

Yes. 

Heading  of  hostile  ships. 

'Snail  trail'  or  history  of  ships  movement. 

Weapon  systems  of  the  hostiles.  If  there  was  full  coverage,  and  I  was  forced  to  enter  a 
detection  range.  I'd  want  to  know  the  weapon  capabilities  of  the  hostiles  so  I  can  make  an 
informed  decision  where  to  fly  with  the  lowest  probability  of  being  shot  down. 
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Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  Improved  useablllty? 

Zooming  and  panning  is  clunky. 

Labels  on  hostile  sites  to  identify  the  emitter. 

A  communications  overlay. 

Terrain  information. 

Summary:  Operator  5  didn't  get  detected  and  reached  destination  in  approx  9:00  mins 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  6 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No. 

Did  you  feel  that  you  had  enough  Information  about  the  detected  radars? 

Yes. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes 

Is  there  additional  Information  that  could  have  made  your  job  easier? 

More  UAVs.  The  UAV  provide  good  feedback. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  Improved  useablllty? 

No.  It  was  good. 

Summary:  Operator  6  didn't  get  detected  and  reached  destination  in  approx  9:30  mins 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  7 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No. 

Did  you  feel  that  you  had  enough  Information  about  the  detected  radars? 

Definitely. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Definitely. 

Is  there  additional  Information  that  could  have  made  your  job  easier? 

No.  I  would  like  the  helicopter  to  go  faster. 

Do  you  feel  the  tools  were  useful  and  useable? 

Definitely. 

What  could  be  changed  to  Improved  useablllty? 
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No.  Perhaps  some  weapons  to  take  out  the  radars. 

Summary:  Operator  7  was  not  detected  and  reached  destination  in  approx.  9:17  mins. 

Answers  to  Questions  for  ELEXSA  Enhanced  Lama  Operator  8 

Do  you  think  you  were  vulnerable  to  detection  by  any  radars? 

No. 

Did  you  feel  that  you  had  enough  information  about  the  detected  radars? 

Yes. 

Did  you  feel  you  had  enough  time  to  react  to  new  radar  detections? 

Yes. 

Is  there  additional  information  that  could  have  made  your  job  easier? 

An  arrow  indicating  the  heading  of  the  helicopter. 

Do  you  feel  the  tools  were  useful  and  useable? 

Yes. 

What  could  be  changed  to  improved  useability? 

No.  Zoom  and  scroll  is  similar  to  other  PC  tools. 

Summary:  Operator  8  was  not  detected  and  reached  destination  in  approx.  9:30  mins. 
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Appendix  B:  The  Fisher  Exact  Probability  Test 

The  Fisher  Exact  Probability  Test  is  a  statistical  test  used  in  the  analysis  of  categorical 
data  where  the  sample  sizes  are  relatively  small.  Categorical  data  is  defined  as  data 
which  belongs  to  exactly  one  of  a  finite  set  of  categories.  The  test  is  used  when 
members  of  two  independent  groups  can  fall  into  one  of  two  mutually  exclusive 
categories.  In  the  ELEXSA  Human  in  the  Loop  Experiment,  the  two  independent 
groups  are  the  operators  who  used  Standard  Llama  and  those  that  used  Enhanced 
Llama  and  the  two  mutually  exclusive  categories,  in  the  case  of  the  first  hypothesis,  are 
completing  and  not  completing  the  assigned  mission.  The  Eisher  test  is  used  to 
determine  whether  the  proportion  of  the  members  from  the  independent  groups 
falling  into  each  category  differs  based  on  the  group. 

In  the  Eisher  test,  as  there  are  two  independent  groups  and  two  categories,  a  2x2 
contingency  table  is  used.  In  statistics,  contingency  tables  are  used  to  record  and 
analyse  the  relationship  between  two  or  more  categorical  variables.  A  2x2  contingency 
table  used  to  express  the  relationship  between  the  variables  is  shown  in  Table  B1 
where  the  values  of  the  cells  are  represented  by  the  letters  a,  b,  c  and  d,  the  totals  across 
rows  and  columns  are  referred  to  as  marginal  totals,  and  the  grand  total  is  represent  by  n. 


Table  Bl.  A  2x2  contingency  table. 


Standard 

Llama 

Enhanced 

Llama 

Total 

Completed 

a 

b 

a  +  b 

Not 

completed 

c 

d 

c  +  d 

Total 

a  +  c 

b  +  d 

n 

If  there  were  no  systematic  association  between  the  Standard  Llama  and  Enhanced 
Llama  groups  within  the  population,  the  probability  of  observing  any  particular  set  of 
frequencies  a,  b,  c,  d  (i.e.  any  particular  arrangement  of  the  data)  in  a  2x2  contingency 
table,  given  fixed  values  for  the  marginal  totals  a+b,  c+d  and  sample  size  n,  would  be 


given  by 


^  a  +  b^f c  + 


P  = 


V 


a 


A  ^  J 


r 


n 

\^a  +  c  j 
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which  reduces  to 

{a  +  b)\{c  +  d)\{a  +  c)\{b  +  d)\ 
n\a\b\c\d\ 


This  value  is  know  as  Pcutoff . 

By  summing  the  probabilities  of  all  other  matrices  with  the  same  marginal  totals  and 
sample  size  which  are  less  than  or  equal  to  the  Pcuttoff,  a  two  sided  probability  is 
obtained.  If  this  value  is  less  than  the  significance  level  of  0.05,  the  proportion  of 
operators  successfully  completing  the  assigned  mission  is  statistically  significant  i.e. 
there  is  a  statistically  significant  association  between  the  type  of  Llama  used  by  the 
operator  and  the  operator  successfully  completing  the  mission.  Thus  the  null 
hypothesis  can  be  rejected  and  Hypothesis  1  (at  95%  level)  can  be  accepted. 

In  the  case  of  Hypothesis  One,  the  Fisher  Exact  Test  is  a  better  choice  than  the  Chi- 
square  test  because  in  order  to  use  the  Chi  Squared  test,  the  raw  observed  frequencies 
cannot  be  too  small  and  further,  because  some  of  the  mathematical  formulas  used  in 
chi  square  use  division,  no  cell  in  the  table  can  have  an  observed  raw  frequency  of  0. 
Also  the  Chi  squared  test  gives  only  an  approximate  solution,  whereas  the  Fisher  test 
returns  exact  one  tailed  and  two  tailed  p- values. 


32 


DSTO-TR-1924 


Appendix  C:  The  f-test 

The  htest  is  the  most  commonly  used  method  to  assess  whether  the  means  of  two 
groups  are  statistically  different  from  each  other.  For  example,  the  f-test  can  be  used  to 
test  for  a  difference  in  test  scores  between  a  group  of  patients  who  were  given  a  drug 
and  a  control  group  who  received  a  placebo.  Theoretically,  the  t-test  can  be  used  even 
if  the  sample  sizes  are  very  small  (e.g.,  as  small  as  10),  as  long  as  the  variables  are 
normally  distributed  within  each  group  and  the  variation  of  scores  in  the  two  groups  is 
not  reliably  different.  In  simple  terms,  the  t-test  compares  the  actual  difference 
between  the  means  in  relation  to  the  variation  in  the  data  which  is  expressed  as  the 
standard  deviation  of  the  difference  between  the  two  means. 


T  Test  Procedure 

For  the  two  samples,  EL  and  SL,  of  sizes  of  Nel  and  Nsl  respectively,  calculate 
The  mean  pel  and  sum  of  squared  deviates  SSel  of  EL 
The  mean  pbl  and  sum  of  squared  deviates  SSsl  of  SL 


where 


ss  =  s 


N 


Estimate  the  variance  of  the  source  population  as: 


{s  }  = 


(N,,  -  1)  +  (N, 


1) 


Estimate  the  standard  deviation  of  the  sampling  distribution  of  sample-mean 
differences  (the  "standard  error"  of  pel— gsr)  as 


est.  o 

p-p 


N 

SL 
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Calculate  t  as 


^  EL  V^SL 

est.a 


Once  this  value  is  calculated,  it  can  be  used  in  a  look  up  table  of  significance  to  test 
whether  the  ratio  is  large  enough  to  say  that  the  difference  between  the  groups  is  not 
likely  to  be  a  chance  finding.  That  is,  if  a  comparison  between  the  calculated  value  of  t 
to  the  critical  values  of  t  is  performed  and  if  the  calculated  value  of  t  exceeds  the 
tabulated  value,  the  means  are  significantly  different.  The  critical  value  for  a 
hypothesis  test  is  a  threshold  to  which  the  value  of  the  test  statistic  in  a  sample  is 
compared  to  determine  whether  or  not  the  null  hypothesis  is  rejected.  The  critical 
value  for  any  hypothesis  test  depends  on  the  significance  level  at  which  the  test  is 
carried  out,  and  whether  the  test  is  one-sided  or  two-sided  and  on  the  degrees  of 
freedom.  The  level  of  significance  is  set  at  0.5  (which  means  that  5  times  out  of  100,  a 
statistically  significant  difference  between  the  means  even  if  there  was  none  would  be 
found)  and  the  degrees  of  freedom  =  Nel  +  Nsl  -  2.  The  critical  values  are  calculated 
from  the  t-distribution  which  has  the  probability  density  function: 


fit) 


r((v+i)/2) 

v/wr(v/2) 


(i+^vvr"" 


with  V  equal  to  n  -  1.  The  parameter  v  is  the  number  of  degrees  of  freedom.  The 
distribution  depends  on  v,  but  not  p  or  o. 

For  example,  for  a  sample  set  of  12  and  a  sensitivity  of  5%,  the  critical  values  will  be 

±2.0. 
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